Exploring and Visualizing Data¶

Shan-Hung Wu & DataLab
Fall 2023

Exploratory Data Analysis¶

Exploratory Data Analysis (EDA) is an important and recommended first step of Machine Learning (prior to the training of a machine learning model that are more commonly seen in research papers). EDA performs the exploration and exploitation steps iteratively. In the exploration step, you "explore" the data, usually by visualizing them in different ways, to discover some characteristics of data. Then, in the exploitation step, you use the identified characteristics to figure out the next things to explore. You then repeat the above two steps until you are satisfied with what you have learned from the data. Data visualization plays an important role in EDA. Next, we use the Wine dataset from the UCI machine learning repository as an example dataset and show some common and useful plots.

Visualizing the Important Characteristics of a Dataset¶

Let's download the Wine dataset using Pandas first:

As we can see, showing data row-by-row with their column names does not help us get the "big picture" and characteristics of data.

NOTE: pd.read_csv() function returns a pandas.DataFrame object. Pandas Dataframe is an useful "two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes".

Pairwise Joint Distributions¶

We can instead see the joint distribution of any pair of columns/attributes/variables/features by using the pairplot function offered by Seaborn, which is based on Matplotlib:

From the above figures, we can easily see that there is a linear relationship between the "Total phenols" and "Flavanoids" variables. Furthermore, the class label cannot be easily predicted by a single variable. You may have noticed that the figures along the diagonal look different. They are histograms of values of individual variables. We can see that the "Ash" and "Alcalinity of ash" variables are roughly normally distributed.

NOTE: importing the Seaborn library modifies the default aesthetics of Matplotlib for the current Python session. If you do not want to use Seaborn's style settings, you can reset the Matplotlib settings by calling:

>>> sns.reset_orig()

Correlation Matrix¶

Showing the pairwise join distributions may still be overwhelming when we have a lot of variables in the dataset. Sometimes, we can just plot the correlation matrix to quantify the linear relationship between variables. The correlation coefficient between two random variables $a$ and $b$ are defined as:

$$\frac{\mathrm{Cov}(\mathrm{a},\mathrm{b})}{\sqrt{\mathrm{Var}(\mathrm{a})\mathrm{Var}(\mathrm{b})}}.$$

Basically, it is the "normalized" variance that captures the linear relationship of the two random variables, and the values are bounded to the range $[−1,1]$. The correlation matrix $\boldsymbol{R}\in\mathbb{R}^{D\times D}$ of a random vector $\mathrm{x}$ is a square matrix whose each element $R_{i, j}$ denotes the correlation between the attributes $\mathrm{x}_{i}$ and $\mathrm{x}_{j}$ . If we regard data points as the i.i.d. samples of $\mathrm{x}$ , then we can have an estimate $\hat{\boldsymbol{R}}$ whose each element

$$\hat{R}_{i,j}=\frac{\Sigma_{s=1}^{N}(x_{i}^{(s)}-\hat{\mu}_{\mathrm{x}_{i}})(x_{j}^{(s)}-\hat{\mu}_{\mathrm{x}_{j}})}{\sqrt{\Sigma_{s=1}^{N}(x_{i}^{(s)}-\hat{\mu}_{\mathrm{x}_{i}})^{2}}\sqrt{\Sigma_{s=1}^{N}(x_{j}^{(s)}-\hat{\mu}_{\mathrm{x}_{j}})^{2}}}=\frac{\hat{\sigma}_{\mathrm{x}_{i},\mathrm{x}_{j}}}{\hat{\sigma}_{\mathrm{x}_{i}}\hat{\sigma}_{\mathrm{x}_{j}}}$$

is an estimate of the correlation (usually called the Pearson's r) between attribute $\mathrm{x}_i$ and $\mathrm{x}_j$ . Note that if we z-normalize each data point such that

$$z_{i}^{(s)}=\frac{x_{i}^{(s)}-\hat{\mu}_{\mathrm{x}_{i}}}{\hat{\sigma}_{\mathrm{x}_{i}}}$$

for all $i$ . Then we simply have $\hat{\boldsymbol{R}}=\frac{1}{N}\boldsymbol{Z}^\top \boldsymbol{Z}$ , where $\boldsymbol{Z}$ is the design matrix of the normalized data points. We can plot $\hat{\boldsymbol{R}}$ as follows:

The correlation matrix gives a more concise view of the relationship between variables. Some models, such the linear regression, assume that the explanatory variables are linearly correlated to the target variable. A heatmap of correlations can help us select variables supporting this assumption. For example, if we want to predict the "Hue" variable using the linear regression, we may pick the "Malic acid" and "Flavanoids" as the explanatory variables.

NOTE: we could have simply used the NumPy function

  >>>R = np.corrcoef(df.values.T)

to get the estimate $\hat{\boldsymbol{R}}$ of the correlation matrix. We calculate $\hat{\boldsymbol{R}}$ by ourselves using the normalized design matrix $\boldsymbol{Z}$ because we will reuse $\boldsymbol{Z}$ later.

PCA for Visualization¶

PCA reduces the dimension of data points, and has been widely used across different machine learning tasks. One application of PCA is to help the visualization of high-dimensional data, as discussed next.

Principal component analysis finds a projection matrix $\boldsymbol{W}=[w^{(1)}, ..., w^{(k)}] \in \mathbb{R}^{D\times K}$ ,where $w^{i}$ are orthonormal vectors, such that each attribute $z^{pca}_{j} = w^{(j)T}z$ has the maximum variance Var($z^{pca}_{j}$).
This problem can be reduced to solve

$$arg \max_{w^{i}\in \mathbb{R}^{D}} w^{(i)T}Z^TZw^{(i)} ,\hspace{5mu} for \hspace{5mu} i \in [1,\hspace{3mu}K]$$

by Rayleigh's Quotient, the optimal $w^{(i)}$ is given by the eigenvector of $Z^TZ$ (or $\hat{\boldsymbol{R}}$) corresponding to the $i$th largest eigenvalue.

Let's summarize PCA in a few simple steps:

1. Standardize the $D$ -dimensional dataset $\boldsymbol{X}$ , e.g., via the $z$ -normalization, and get $\boldsymbol{Z}$ ;
2. Estimate the covariance matrix $\hat{\boldsymbol{R}}$;
3. Decompose $\hat{\boldsymbol{R}}$ into its eigenvectors and eigenvalues;
4. Select $K$ eigenvectors that correspond to the $K$ largest eigenvalues, where $K$ is the dimensionality of the new feature subspace ($k < d$);
5. Construct a projection matrix $\boldsymbol{W}$ from the top-$K$ eigenvectors;
6. Transform the $D$-dimensional input dataset $\boldsymbol{Z}$ using the projection matrix $\boldsymbol{W}$.

Eigendecomposition¶

Since we already have $\boldsymbol{Z}$ and $\hat{\boldsymbol{R}}$ from the above. We can begin from the step 3:

NOTE: there is an np.linalg.eig() function in NumPy that also eigendecomposes matrices. The difference is that np.linalg.eigh() is optimized for symmetric matrices whose eigenvalues are always real numbers. The numerically less stable np.linalg.eig() can decompose non-symmetric square matrices and returns complex eigenvalues.

Eigenvector Selection¶

In step 4, we need to decide the value of $K$. We can plot the variance explained ratio of each eigenvalue:

$$\frac{\vert\lambda_{j}\vert}{\Sigma_{j=1}^{D}\vert\lambda_{j}\vert}$$

in the descending order to help us decide how many eigenvectors to keep.

The resulting plot indicates that the first principal component alone accounts for 40 percent of the variance. Also, we can see that the first two principal components combined explain almost 60 percent of the variance in the data. Next, we collect the two eigenvectors that correspond to the two largest values to capture about 60 percent of the variance in this dataset. Note that we only chose two eigenvectors for the purpose of illustration, since we are going to plot the data via a two-dimensional scatter plot later in this subsection. In practice, the number of principal components can be determined by other reasons, such as the trade-off between computational efficiency and performance.

Feature Transformation¶

Let's now proceed with the last three steps to project the standardized Wine dataset onto the new principal component axes. We start by sorting the eigenpairs by decreasing order of the eigenvalues:

Next, we pick the first two eigenvectors and form the project matrix $\boldsymbol{W}$:

Finally, we can obtain the compressed dataset by: $$\boldsymbol{Z}^{\text{PCA}}=\boldsymbol{Z}\boldsymbol{W}$$ and visualize it using:

As we can see, the data is more spread along the $x$-axis corresponding to the first principal component than the $y$-axis (second principal component), which is consistent with the explained variance ratio plot that we created in the previous subsection. PCA may save us from examining a lot of pairwise distributions (as shown in the previous section) when the original data dimension $D$ is high. For example, if we see that the data points with different labels can be separated in the space of PCA, then we can simply choose a linear classifier to do the classification.

Finally, let's save the compressed dataset for future use.

Assignment¶

Here's a generated dataset, with 3 classes and 15 attributes. Your goal is to reduce data dimension to 2 and 3, and then plot 2-D and 3-D visualization on the compressed data, respectively.

You can see here for information about plotting 3D graph

From this assignment, you can see the different results between different numbers of principle components chosen.

Requirements:¶

• Submit to eeclass with your code named Lab02_{student-id}.ipynb (e.g. Lab02_109069999.ipynb).
• The code file should only contain the Assignment part.
• Remember to save the file after you rendered the output images in your notebook.
• Deadline: 2024-12-20 (Fri) 23:59.